feat(go/ai): added DefineMiddleware (Middleware V2)#4464
feat(go/ai): added DefineMiddleware (Middleware V2)#4464
DefineMiddleware (Middleware V2)#4464Conversation
Summary of ChangesHello @apascal07, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the extensibility of the AI generation pipeline by introducing a comprehensive middleware system. It moves beyond simple model-level interception to offer granular control over the entire generation lifecycle, including tool usage and the iterative generation loop. This change provides a powerful mechanism for developers to inject custom logic, such as logging, metrics, or content filtering, at various critical points within the AI workflow, making the system more adaptable and observable. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a new, more powerful Middleware system for Go, which is a great enhancement. It adds hooks for Generate, Model, and Tool execution stages, deprecating the older ModelMiddleware. The implementation is solid and includes good test coverage. I've identified one bug related to data loss in tool responses and an opportunity to deduplicate some code. Overall, this is a well-executed feature.
MiddlewareMiddleware
MiddlewareDefineMiddleware (Middleware V2)
Co-authored-by: Pavel Jbanov <pavelj@google.com>
0aa7385 to
ff1798c
Compare
MichaelDoyle
left a comment
There was a problem hiding this comment.
LGTM from a genkit-tools perspective.
Rework the Go middleware primitives introduced in PR #4464 to collapse configuration and behavior into a single "config struct is the middleware" model and remove the descriptor/factory/prototype scaffolding. - Drop the Middleware interface (Name/New/WrapGenerate/WrapModel/WrapTool/Tools) and the BaseMiddleware embedding helper. Introduce Hooks as a plain struct of optional hook func fields (WrapGenerate, WrapModel, WrapTool, Tools); nil hooks pass through. - Repurpose Middleware as an interface with just Name() + New(ctx), which a user-facing config struct implements directly. Passing a config value to WithUse runs its New on the local fast path with no registry lookup, so pure-Go code works without plugin registration. - NewMiddleware[M](description, prototype) captures the typed prototype in a closure stored on MiddlewareDesc.buildFromJSON, preserving unexported plugin-level state across JSON-dispatched calls via value-copy. - MiddlewareDesc returns to being the shared schemas.config-generated type with the private factory added via the existing `field` directive. - Rename MiddlewarePlugin.ListMiddleware to Middlewares to align with the upcoming V2 naming conventions. - Replace Inline with MiddlewareFunc, a canonical Go adapter type that satisfies Middleware for ad-hoc closure-based middleware. - Add genkit.DefineMiddleware and genkit.LookupMiddleware wrappers with complete godoc matching the DefineTool/LookupTool style. Fixes carried over from the initial review: - Preserve MultipartToolResponse.Content through the resume path in handleResumedToolRequest (previously dropped). - Change WrapTool return type to *MultipartToolResponse so metadata and content flow through without an out-of-band capture hack. - Reject duplicate middleware-contributed tool names explicitly in GenerateWithRequest instead of panicking at registry registration. - Build the WrapGenerate, WrapModel, and WrapTool hook chains once per GenerateWithRequest rather than rebuilding them on every tool-loop turn. - Export NewToolInterruptError so WrapTool hooks can interrupt tools without constructing a ToolContext. Tests rewritten against the new shape and expanded to cover: plugin-state value-copy, call-level state isolation, MiddlewareFunc adapter, nil hooks, stream chunk accumulation, tool contribution, duplicate-tool rejection, factory error propagation, WrapTool interrupts, per-iteration WrapGenerate, and metadata round-trip through WrapTool. All green under -race.
|
@pavelgj Minor but pretty great improvements to the middleware abstraction in case you want to review it again. The PR description covers it all. |
Adds a
Middlewareinterface for wrapping generation, model calls, and tool execution with composable hooks, plus the registration and plugin plumbing needed to expose middleware to the Dev UI and cross-runtime tooling. Built-in implementations (retry, fallback, tool approval, filesystem, skills) ship in a follow-up PR.The core mental model: a middleware is a config struct with two methods.
Name()returns its stable registered identifier andNew(ctx)produces a per-call bundle of hook functions.Examples
Attaching middleware to a Generate call
Middleware is attached per call with
ai.WithUse. The chain applies outer-to-inner, soai.WithUse(A, B)expands toA { B { model } }:No plugin registration is required for pure-Go use.
WithUsecalls the config'sNewdirectly on the local fast path. Registration is what makes middleware visible to the Dev UI and callable from other runtimes.Defining your own middleware
Implement
ai.Middlewareby providing two methods on a config struct.Name()returns the registered identifier (a stable constant read from the zero value of the type).New(ctx)returns an*ai.Hooksvalue with any ofWrapGenerate,WrapModel,WrapTool, andToolspopulated. Nil fields pass through.Per-call state is closure-captured inside
New(which runs once perGenerateinvocation). Plugin-level state belongs on unexported fields of the config struct; the plugin'sMiddlewares()method sets those fields on a prototype that value-copies across JSON-dispatched calls to preserve them.Ad-hoc inline middleware
For one-off middleware that does not need Dev UI visibility or a named type, use
ai.MiddlewareFunc:Plugin-provided middleware
Plugins advertise middleware by implementing
MiddlewarePlugin.Middlewaresis called duringgenkit.Init, and the returned descriptors become visible to the Dev UI and available for lookup by name. Any unexported fields on the prototype (likeclientbelow) are preserved across Dev UI and cross-runtime invocations via value-copy in the descriptor's build closure.API Reference
ai.Middlewareinterfaceai.HooksbundleWrapGeneratewraps each iteration of the tool loop (invokedN+1times for a generate withNtool turns).WrapModelwraps each model API call.WrapToolwraps each tool execution and may run concurrently for parallel tool calls.Toolscontributes additional tools to register for the generation.Hook parameters
Registration
Matching conveniences exist in the
genkitpackage:Application code typically uses
genkit.DefineMiddlewareto register middleware it owns directly. Plugin authors typically useai.NewMiddlewareand return descriptors fromMiddlewarePlugin.Middlewares();genkit.Initregisters them during plugin setup.Inline adapter
Attaching to a call
Plugin integration
Design notes
WithUseis the same type whoseNewmethod the runtime invokes. Plugin-level state rides on unexported fields and is preserved across JSON dispatch by value-copy inside the descriptor's build closure. Per-call state lives in closures captured byNew.NewMiddlewarecaptures the typed prototype in a closure that unmarshals JSON on a value copy (preserving unexported fields, sharing pointers). NoBaseMiddlewareembedding or field-by-field copy required.GenerateWithRequestand reused across every tool-loop iteration instead of being rebuilt per turn.WithUse(Retry{...})invokesRetry.New(ctx)directly on the fast path; the registry is consulted only for JSON-dispatched invocations from the Dev UI or cross-runtime callers.Other changes
ai.ModelMiddleware/ai.WithMiddlewareAPI is preserved and marked deprecated. New code should preferai.Middleware/ai.WithUse, which addsWrapGenerateandWrapToolhooks and supports dynamically injected tools viaHooks.Tools.MultipartToolResponse.Contentnow flows through the resume path (previously dropped inhandleResumedToolRequest).WrapToolreturns*MultipartToolResponseso metadata and content round-trip without an out-of-band capture.GenerateWithRequestrather than panicking during registry registration.NewToolInterruptErroris exported soWrapToolhooks can interrupt tools without constructing aToolContext.genkit-toolsandpy/packages/genkitkeep the middleware reference type consistent across the JS, Go, and Python runtimes.Retry,Fallback,ToolApproval,Filesystem, andSkills.